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ABSTRACT 



This paper describes the "Urban School Key Indicators of 
Science and Mathematics Education, " a four-volume publication designed to 
visualize the educational reform progress of 21 urban school districts in the 
National Science Foundation's Urban Systemic Initiative (USI) program 
launched in 1994. The USI program is intended to be a catalyst for 
large-scale educational change affecting standards, curriculum, assessment, 
professional development, partnerships, and convergence of intellectual and 
fiscal resources, with constant attention to improving student achievement. 
The "Urban School Key Indicators of Science and Mathematics Education" 
culminates the first year and a half. effort to collect and compile 
multiple-year reform progress data across the 21 sites using the Key 
Indicator Data System, which includes quantitative and qualitative data 
templates. Quantitative data include information on student demographics, 
academics, test scores, and teacher professional development and district and 
USI fund utilization. The Urban School Key Indicators report presents the 
quantitative and qualitative multiple-year reform data in a condensed graphic 
and text format. It provides a rationale for establishing a consortium for 
the urban school key indicators data warehouse designed to assist urban 
school districts, stakeholders, and research communities. (SM) 
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Background and Objective 

The “ Urban School Key Indicators of Science and Mathematics Education ” is a four- 
volume publication designed to visualize the educational reform progress of 21 urban 
school districts in National Science Foundation’s Urban Systemic Initiative (USI) 
program launched in 1994. Each urban school district is implementing a five-year 
concerted system-wide effort to promote standards-based reform focused on 
mathematics, science, and technology (MST). The USI program has been designed as a 
catalyst for large-scale educational change affecting standards, curriculum, assessment, 
professional development, partnerships, and convergence of intellectual and fiscal 
resources, with constant attention to improving student achievement. 

Over the course of its three SI programs (State, Urban, and Rural), NSF has developed a 
theoretical structure for systemic reform that is based on six “drivers” including four 
process drivers (curriculum, policy, resources, and broad-based support) and two student 
outcome drivers (student achievement, improvement of historically underserved), as well 
as a number of cross-cutting issues such as equity, quality, scaling up, coordination and 
organization. 

Systemic Research, Inc. received a three year grant, entitled “ How Reform Works: An 
Evaluative Study of NSF’ s Urban Systemic Initiatives, ” to explore the impact of the USI 
program on student achievement and the learning infrastructure in urban school districts 
by examining causal inferential models among the process drivers (independent 
variables) and outcome drivers (dependent variables). Our evaluative study team 
published and disseminated the first two monographs to identify critical variables in 
education reform and classroom practices (Ref. 1 & 2). 



Methodology and Framework 

The Urban School Key Indicators of Science and Mathematics Education culminates the 
first year and a half effort to collect and compile multiple-year reform progress data 
across 21 USI sites using the Key Indicator Data System (KIDS) designed and 
implemented by Systemic Research. KIDS consists of two electronic data collection 
instruments: quantitative data template (K-l) and qualitative data template (K-2). 
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K-l consists of 
Table A: 
Table B: 
Table C: 
Table D: 
Table E: 
Table F: 
Table G: 
Table H: 
Table I: 
Table J: 
Table K: 



1 1 sections of input templates for multi-year data collection: 

District-Wide Student Demographics and Selected Statistics 

Math and Science Gate-Keeping Course Enrollment & Completion 

12th Grade Students Graduation Data Summary 

High School Graduation Requirements 

AP Mathematics and Science Test Results 

SAT-I : Mathematics and Verbal Reasoning Test Results 

ACT: Mathematics, Science Reasoning, and English Test Results 

Assessment Test Results 

Mathematics/Science Teacher Certificates and Professional Development 
District and USI Fund Utilization 
Definition of School Statistics Terminologies 



Table H has been tailored to each school district to accommodate selected types of 
mathematics and science assessment instruments. Our evaluative study team established 
collaborative research agreements with the Educational Testing Service (ETS), the 
College Board, and ACT, Inc., respectively, and obtained six year of SAT-I, AP, and 
ACT test results directly from each agency. 

K-l is a Microsoft ® Excel-based, menu-driven, user-friendly instrument designed to 
provide automatic data verification and output generating capabilities. The second part 
of K-l automatically generates a series of output figures linking longitudinal input tables 
for review and analysis purpose. 

K-2 is another Microsoft ® Excel-based instrument designed to collect qualitative key 
indicators over the project period which complements K-l. K-2 includes descriptive 
questionnaires regarding the SI drivers: policy, standards-based curriculum and 
instruction, professional development, standards-based assessment, district leadership and 
partnerships, and accountability. Our evaluative study team initiated K-2 data collection 
by extracting available data from each site’s Performance Effectiveness Reviews, Annual 
Reports, and Common Data Elements (CDE) submitted to NSF. 



• Cohort/Scale-up Approach 

• Primary Decision Making Body for Various School District Agenda 

• Policies Promoting Equal Access by All Students in High Quality Education 

• Policies Impacting the Enrollment of Students in Challenging Math and Science 
Courses 

• Policies Relevant to Curriculum 

• Standards-based Curriculum and Instruction 

• Policies Relevant to Teacher Qualifications 

• Professional Development Policies and Practices 

• Policies Relevant to Standards-based Assessments 

• USI Leadership, Governance, and Management 

• Partnerships 
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To promote collaborative research environments and to enhance data integrity, our 
evaluative study team sponsored an annual KIDS workshop inviting USI project 
directors, data managers, and local evaluators along with NSF program directors. After 
many iterations of data collection between each USI site and our study team, 21 sets of 
K-l and K-2 could finally merged into a four-volume, 490 page report, “ Urban School 
Key Indicators of Science and Mathematics Education. ” The first three volumes present 
the progress of 21 USI districts by cohort including program data, project summary, and 
project goals along with web site information: 

• Volume I - Cohort 93 (Baltimore, Chicago, Dallas, Detroit, El Paso, Miami- 
Dade, New York, and Phoenix) 

• Volume It- Cohort 94 (Cleveland, Columbus, Fresno, Los Angeles, Memphis, 
New Orleans, and Philadelphia) 

• Volume in - Cohort 95 and 97 (Milwaukee, St. Louis, San Antonio, San Diego, 
Atlanta, and Jacksonville) 

The Volume IV Key Indicator Report presents USI overall progress data by aggregating 
student achievement data (gate-keeping course enrollment and completion, SAT-I, AP, 
ACT), 12 th grade graduation data, assessment data, and teacher and professional 
development data. The overall progress report also includes tabulation of math and 
science assessment instruments, graduation requirements, and district and USI fund 
allocations. 



Conclusion 

The “ Urban School Key Indicators of Science and Mathematics Education ” has been 
published as a component of the NSF granted evaluative study “How Reform Works: An 
Evaluative Study of NSF’s Urban Systemic Initiatives.” The report has been 
disseminated by full-color hard copy, as well as a CD-ROM version and a PDF-based 
web version. 

Our evaluative study team uses the Urban School Key Indicator report as a first stepping- 
stone to explore the inferential causal model of the best practices in urban school reform. 
The Key Indicator report is unique and one-of-a-kind in terms of presenting both 
quantitative and qualitative multiple-year reform data in a condensed graphic and text 
format. The Urban School Key Indicator report was fully utilized by NSF program 
directors, USI PI/PDs, data managers, local evaluators, as well as other urban SI research 
groups. The Urban Key Indicators report provides a rationale to establish a consortium 
for the urban school key indicators data warehouse designed to assist urban school 
districts, stakeholders, and research communities. 

1. Ware, M., Richardson, L., & Kim, J. (1999). What Matters in Urban School Reform. Study 
Monograph No. 1- “How Reform Works: An Evaluative Study of NSF’s Urban Systemic 
Initiatives.” Norwood, MA: Systemic Research, Inc. 
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2. Blank, R., Kim, J., & Smithson, J. (2000). Survey Results of Urban School Classroom Practices 
in Mathematics and Science: 1999 Report. Study Monograph No. 2- “How Reform Works: An 
Evaluative Study of NSF’s Urban Systemic Initiatives.” Norwood, MA: Systemic Research, Inc. 



O 

ERIC 



4 



6 







t I 

U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 

National Library of Education (NLE) 

Educational Resources Information Center (ERIC) 

REPRODUCTION RELEASE ttOovun, 

(Specific Document) 




I. DOCUMENT IDENTIFICATION: 



Title: Sch*>l feof ?&"**»* , 

^ia <-/%£. eu r //hfa* 4ys*’ e * h,Ct "ZblbA+ic^. 






Authors) 



: xJfrSfh -A KjJ>i , £tg<gl 



Corporate Source: 



~/<z. tee&Arz* f i , 



Publication Date: 

2000 



II. REPRODUCTION RELEASE: 

In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced in the 
monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche, reproduced paper copy, 
and electronic media, and sold through the ERIC Document Reproduction Service (EDRS). Credit is given to the source of each document, and, if 
reproduction release is granted, one of the following notices is affixed to the document. 



If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following three options and sign at the bottom 
of the page. 




Level 1 



s. 



Level 2A 



□ 



Level 2B 



□ 



Check here for Level 1 release, permitting 
reproduction and dissemination in microfiche or other 
ERIC archival media (e g., electronic) and paper 
copy. 



Check here for Level 2A release, permitting 
reproduction and dissemination in microfiche and in 
electronic media for ERIC archival collection 
subscribers only 



Check here for Level 2B release, permitting 
reproduction and dissemination in microfiche only 



Documents will be processed as indicated provided reproduction quality permits. 

If permission to reproduce is granted, but no box is checked, documents will be processed at Level 1 



Sign 
here i,-* 



n/Mse 




I hereby grant to the Educational Resources Information Center (ERIC) nonexclusive pemiission to reproduce and disseminate this document 
as indicated above. Reproduction from the ERIC microfiche or electronic media by persons other than ERIC employees and its system 
contractors requires permission from the copyright holder. Exception is made for non-profit reproduction by libraries and other service agencies 
to satisfy information needs of educators in response to discrete inquiries. 



Signature: 







3^ 



Printed Name/Position/Title: 

J/tftu J. Kiru 



//*-< 



<5/ ' 



OrganizatiorVAddress: 



/ So 44 A o a. 0 o 2 



T T7?n 2.7 2-43*° F ^Xnll 

Co, r* 4/1/0 i 



(over) 




III. DOCUMENT AVAILABILITY INFORMATION (FROM NON-ERIC SOURCE): 

If permission to reproduce is not granted to ERIC, or, if you wish ERIC to cite the availability of the document from another source, please 
provide the following information regarding the availability of the document. (ERIC will not announce a document unless it is publicly 
available, and a dependable source can be specified. Contributors should also be aware that ERIC selection criteria are significantly more 
stringent for documents that cannot be made available through EDRS.) 




IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS HOLDER: 

If the right to grant this reproduction release is held by someone other than the addressee, please provide the appropriate name and 
address: 




V. WHERE TO SEND THIS FORM: 



Send this form to the following ERIC Clearinghouse: 

University of Maryland 

ERIC Clearinghouse on Assessment and Evaluation 
1129 Shriver Laboratory 
College Park, MD 20742 
Attn: Acquisitions 



However, if solicited by the ERIC Facility, or if making an unsolicited contribution to ERIC, return this form (and the document being 
contributed) to: 

ERIC Processing and Reference Facility 
1100 West Street, 2 nd Floor 
Laurel, Maryland 20707-3598 



EFF-088 (Rev. 9/97) 



Telephone: 301-497-4080 
Toll Free: 800-799-3742 
FAX: 301-953-0263 
e-mail: ericfac@ineted.gov 
WWW: http://ericfac.piccard.csc.com 



O 

ERLC 



